Optimal Precoder Design for Non-Regenerative MIMO Cognitive Two-Way Relay Systems with Underlay Spectrum Sharing

We study the optimal precoder design for a MIMO cognitive two-way relay system with underlay spectrum sharing. The system consists of two secondary users (SUs) and one relay station (RS). We jointly optimize the precoders for SUs and RS with perfect and imperfect channel state information (CSI) between SUs/RS and the primary user, where our design approach is based on the alternate optimization method. For the perfect CSI case, we derive the optimal structure of the RS precoding matrix, which generalizes the result for single-antenna SUs and helps to reduce the search complexity. We develop gradient projection (GP) algorithm to calculate the optimal RS precoder numerically. When the RS precoder is given, we propose a fast algorithm based on generalized water-filling theorem to compute the optimal SU precoders. For the imperfect CSI case, we derive equivalent conditions for the interference power constraints and convert the robust SU precoder optimization into the form of semi-definite programming. As for the robust RS precoder optimization, we relax the interference power constraint related with the RS precoder to be convex and then the GP algorithm can be applied. Finally, simulation results demonstrate the effectiveness of the proposed schemes.